OCDBT support for graphene supervoxel splitting#631
Open
OCDBT support for graphene supervoxel splitting#631
Conversation
using l2cache when available for fast and more accurate find path check for l2cache existing is now cached and only checked when necessary
…auth can catch terms of service errors and prompt the user to agree to the terms of service fixed lint errors and minor cleanup
…perspectiveView, sliceView) cleaned up code, addressed PR comments
…ier, allowing multiple contexts to be modified
…rrent active layer's annotation list
displaying annotation tags in annotation list
…p the original bug but it got merged into commit ea732e8
… property to SegmentationUserLayer
…r backwards compatibilty
…nt (including segment id 0) hidden (same as off color) as well as making them appear to have no highlight effect by setting the highlight color to be the same as the off color (other than the focus segment)
…ol to allow place line tool to function. Swap back when tool is unloaded. Also swapping back to the merge tool layer's legacy tool
feat: Add 0.6 ome zarr rotation transform support chore: Update zarr tests feat: Add mapAxis support chore: Add mapzAxis example feat: Add sequence support feat: Make neuroglancer constraints explicit fix: Correct mapAxis implementation chore: Add identity example feat: Update rotation test feat: Update affine example and test feat: Add input / output validation feat: Add proper multiscale test chore: Remove multiscale tests chore: run formatter refactor: extract scale function to different file feat: run transform from ome to model transform feat: pull out user transform as model transform fix: more general base transform back out feat: more complete scale calculation feat: add new affine scale method and use in frontend refactor: small clean up of ome zarr fix(test): update pyton tests for new transform handling test: update test data to use symlink to reduce size fix: correct map axis matrix filling refactor: clean up some code and tests test: add a number of parsing tests test: add multiscale test chore: remove accidental log from tests docs: update code comments test: clarify it and order for affine test: add example chore: formattign chore: add back newline for clean diff fix: remove unsupported zstandard codec from test data (should be zstd) revert: change zstd decoding and resolving back to master branch state fix: remove 0.6-dev2 version
added appspot and state share config bumped appengine runtime to python312 since "Runtime python27 is end of support and no longer allowed" removed theadsafe - "This field is not supported with runtime [python312] and can safely be removed." removed skip_files (no longer supported) "api_version" field is not allowed in runtime python312. forgot to add .gcloudignore (replacement of skip_files removed "on pull request"
Adds invalidateOcdbtCaches() so consumers can clear the three metadata caches after a server-side OCDBT mutation, forcing the next read to resolve a fresh root. Also removes the per-instance root memoization on OcdbtKvStore -- it was redundant with the ocdbt:version SimpleAsyncCache and prevented invalidation from taking effect.
Adds ocdbt_seg / ocdbt_path graph-info fields. When ocdbt_seg is set, segmentation volume reads route through a per-scale OCDBT pipeline URL (scales auto-discovered via list()). Non-OCDBT scales are filtered from getSources() so the graphene layer only shows data available in the fork. After a multicut, invalidates the OCDBT metadata caches via RPC so split supervoxels become visible without a manual reload. Also skips the "supervoxel already selected" guard in the multicut tool when ocdbt_seg is active, since SV splits require selecting the same supervoxel on both sides of the cut.
Adds a base kvstore driver that routes reads/stats to different
backing stores based on per-layer matchers (base / exact / prefix),
matching the semantics of tensorstore's kvstack driver. Last-match
wins on overlaps.
URL form is `kvstack:<percent-encoded-JSON-spec>[/<path>]`, with
the JSON matching tensorstore's `{"layers":[{base,exact|prefix}]}`
shape so specs are portable.
Used as the base under OCDBT for pcg v3 fork layouts (next commit).
Replaces the client-side construction of the OCDBT URL from ocdbt_seg + ocdbt_path with a single new info-JSON field `graph.ocdbt_kvstore_spec` that carries the full tensorstore kvstore spec (ocdbt wrapping kvstack) verbatim from pcg. The client unwraps the spec, URL-encodes its `.base` (the kvstack layers) as `kvstack:<percent-encoded-json>`, and appends `|ocdbt:` to get the neuroglancer pipeline URL. OCDBT-level `config` and `*_data_prefix` fields in the spec are ignored on reads per tensorstore docs. Presence of ocdbt_kvstore_spec is now the OCDBT-enabled signal; absent spec bypasses the kvstack path entirely so legacy v2 graphene layers are unaffected. All remaining `ocdbtSeg` checks swap to `ocdbtKvstoreSpec === undefined` / presence.
…ncCache.invalidateAll Adds a new SimpleAsyncCache.invalidateAll() helper and uses it to collapse the three hand-rolled invalidation loops in invalidateOcdbtCaches into three one-liners. Drops the unused baseUrl parameter: invalidation was already whole-context broad (every OCDBT database in the viewer is flushed), and the param was never consulted. RPC payload and handler shrink accordingly. Adds a comment explaining why the inline stub factories throw: real factories are registered by prior getManifest/getBtreeNode/getRoot calls, so memoize.get returns the existing SimpleAsyncCache without touching the stubs. No changes to upstream OCDBT functions (getManifest, getBtreeNode, getRoot); invalidateOcdbtCaches was added by us and stays the only OCDBT-side entry point.
Two small fixes to avoid footguns:
- formatKvStackUrl now percent-encodes the key portion, not just the
JSON. Keys containing `?`, `#`, or `%` previously produced URLs that
either failed to parse or round-tripped to a different value, since
parseKvStackUrl already decodes the path via decodeURIComponent.
- validateKvStackSpec now rejects empty `layers`, empty `base`, and
empty `prefix`. Empty layers silently routed nothing; empty prefix
degenerated to a catch-all (`"".startsWith("")` is true), shadowing
any preceding `base` layer in unobvious ways. Callers should use an
explicit base-only layer for catch-all routing.
parseGrapheneMultiscaleVolumeInfo calls list() on the OCDBT URL to discover which scales are backed by the fork. Previously any failure (transient network, auth, misconfigured spec) propagated as an opaque error from deep in kvstore; the user saw "read failed" with no hint that OCDBT setup was the root cause. Wrap the list() call in a try/catch that rethrows with the OCDBT URL and a note that the graphene layer cannot render without the scale list. No behavior change on the happy path.
…outing context Wraps each kvstack read/stat in a bounded retry loop so that transient failures don't get latched into the OCDBT metadata caches (which `asyncMemoizeWithProgress` caches permanently for the page lifetime). The retry handles HttpError status 0 (network/CORS) and 502 -- 429/503/504 are already retried inside fetchOk so we don't double-cover them. Backoff reuses the existing pickDelay helper from util/http_request.ts (jittered exponential, no new magic numbers). Sleeps are abort-aware so navigating away cancels them cleanly. After retries are exhausted (or on a non-retryable error), wraps with a message naming the matched layer (base / exact:KEY / prefix:PREFIX) and the backing URL, with the original error in `cause`. Makes "the fork manifest 404'd" obvious instead of "some random GCS read failed".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for reading graphene segmentation backed by an OCDBT database with a kvstack overlay: an immutable base layer holds source watershed data shared across graphs, and per-graph "fork" layers hold edits. Readers must route through the full stack to see the fork's view — reading base alone misses edits, reading the fork alone misses everything that hasn't been edited.
base/exact/prefixmatchers, last-match-wins, lazy layer resolution). URL formkvstack:<percent-encoded-JSON>[/<path>].graph.ocdbt_kvstore_specfrom the server and builds akvstack:...|ocdbt:pipeline URL; per-scale routing sends OCDBT-backed scales through the fork and hides non-OCDBT scales so stale precomputed data can't leak in.ocdbt:manifest/btree/versioncaches so the next read resolves a fresh B+tree root without a manual reload.SimpleAsyncCache.invalidateAll()and validator/encoding hardening on the kvstack URL; surface OCDBT scale-list failures with the offending URL instead of an opaque kvstore error.